Methods and Apparatus for Full-Resolution Light-Field Capture and Rendering

ABSTRACT

Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to flats captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

CONTINUATION DATA

This application is a divisional of U.S. application Ser. No.12/474,112, filed May 28, 2009, which is a continuation-in-part of U.S.patent application Ser. No. 12/271,389 entitled “Methods and Apparatusfor Full-Resolution Light-Field Capture and Rendering” filed Nov. 14,2008, the content of which is incorporated by reference herein in itsentirety, which claims benefit of U.S. Provisional Application Ser. No.61/023,036 entitled “Full-resolution Lightfield Rendering” filed Jan.23, 2008, the content of which is also incorporated by reference hereinin its entirety.

BACKGROUND Description of the Related Art

In a conventional camera, the main lens maps the 3D world of the sceneoutside camera into a 3D world inside camera. FIG. 1 illustrates imagingin a conventional camera. “Inside world” represents inside the camera.The shaded oval regions represent the order of depths in the outsideworld, and the corresponding depths inside the camera. One particularimage plane inside the camera is shown. The mapping of the 3D world ofthe scene outside camera into a 3D world inside camera is governed bythe lens equation:

${\frac{1}{A} + \frac{1}{B}} = \frac{1}{F}$

where A and B are respectively the distances from the lens to the objectplane and from the lens to the image plane. This equation is normallyused to describe the effect of a single image mapping between two fixedplanes. In reality, however, the lens equation describes an infinitenumber of mappings—it constrains the relationship between, but does notfix, the values of the distances A and B. That is, every plane in theoutside scene (which is described as being at some distance A from theobjective lens) is mapped by the objective lens to a corresponding planeinside of the camera at a distance B. When a sensor (e.g., conventionalfilm, a charge-coupled device (CCD), etc.) is placed at a distance Bbetween F and ∞ (infinity) inside the camera, the sensor captures anin-focus image of the corresponding plane at A that was mapped from thescene in front of the lens.

Conventional cameras render a three-dimensional scene onto atwo-dimensional sensor. During operation, a conventional digital cameracaptures a two-dimensional (2-D) image representing a total amount oflight that strikes each point on a photosensor within the camera.However, this 2-D image contains no information about the direction ofthe light that strikes the photosensor. The image captured by aconventional camera essentially integrates the radiance function overits angular portion, resulting in a two-dimensional intensity as afunction of position. The angular information of the original radianceis lost. Thus, conventional cameras fail to capture a large amount ofoptical information.

Light-Field or Radiance Capturing Cameras

In contrast to conventional cameras, light-field, or radiance capturing,cameras sample the four-dimensional (4-D) optical phase space orlight-field, and in doing so capture information about the directionaldistribution of the light rays. This information captured by light-fieldcameras may be referred to as the light-field, the plenoptic function,or radiance. In computational photography, a light-field is a 4-D recordof all light rays in 3-D. Radiance describes both spatial and angularinformation, and is defined as density of energy per unit of area perunit of stereo angle (in radians). A light-field camera capturesradiance; therefore, light-field images originally taken out-of-focusmay be refocused, noise may be reduced, viewpoints may be changed, andother light-field effects may be achieved.

Light-fields, i.e. radiance, may be captured with a conventional camera.In one conventional method, M×N images of a scene may be captured fromdifferent positions with a conventional camera. If, for example, 8×8images are captured from 64 different positions, 64 images are produced.The pixel from each position (i, j) in each image are taken and placedinto blocks, to generate 64 blocks. FIG. 2 illustrates an example priorart light-field camera, or camera array, which employs an array of twoor more objective lenses 110. Each objective lens focuses on aparticular region of photosensor 108, or alternatively on a separatephotosensor 108. This light-field camera 100 may be viewed as acombination of two or more conventional cameras that each simultaneouslyrecords an image of a subject on a particular region of photosensor 108or alternatively on a particular photosensor 108. The captured imagesmay then be combined to form one image.

FIG. 3 illustrates an example prior art plenoptic camera, another typeof radiance capturing camera, that employs a single objective lens and amicrolens or lenslet array 106 that includes, for example, about 100,000lenslets. In a conventional plenoptic camera 102, lenslet array 106 isfixed at a small distance (˜0.5 mm) from a photosensor 108, e.g. acharge-coupled device (CCD). In conventional plenoptic cameras, themicrolenses are placed and adjusted accurately to be exactly at onefocal length f from the sensor 108. This is done by placing the array ofmicrolenses at distance f from the sensor, where f is the focal lengthof the microlenses. Another way to say this is that, for themicrolenses, f is chosen to be equal to the distance to the photosensor108. In other words, the microlenses are focused on infinity, which isessentially equivalent to focusing the microlenses on the main lens 104,given the large distance of the microlenses to the main lens relative tothe focal length of the microlenses. Thus, the raw image captured withplenoptic camera 102 is made up of an array of small images, typicallycircular, of the main lens 108. These small images may be referred to asmicroimages. However, in conventional plenoptic camera 102, eachmicrolens does not create an image of the internal world on the sensor108, but instead creates an image of the main camera lens 104.

The lenslet array 106 enables the plenoptic camera 102 to capture thelight-field, i.e. to record not only image intensity, but also thedistribution of intensity in different directions at each point. Eachlenslet splits a beam coming to it from the main lens 104 into rayscoming from different locations on the aperture of the main lens 108.Each of these rays is recorded as a pixel on photosensor 108, and thepixels under each lenslet collectively form an n-pixel image. Thisn-pixel area under each lenslet may be referred to as a macropixel, andthe camera 102 generates a microimage at each macropixel. The plenopticphotograph captured by a camera 102 with, for example, 100,000 lensletswill contain 100,000 macropixels, and thus generate 100,000 microimagesof a subject. Each macropixel contains different angular samples of thelight rays coming to a given microlens. Each macropixel contributes toonly one pixel in the different angular views of the scene; that is,only one pixel from a macropixel is used in a given angular view. As aresult, each angular view contains 100,000 pixels, each pixelcontributed from a different macropixel. Another type of integral orlight-field camera is similar to the plenoptic camera of FIG. 3, exceptthat an array of pinholes is used between the main lens and thephotosensor instead of an array of lenslets.

FIG. 4 further illustrates an example prior art plenoptic camera model.In conventional plenoptic camera 102, the microlens-space system swapspositional and angular coordinates of the radiance at the microlens. Forclarity, only the rays through one of the microlenses are illustrated.The conventional optical analysis of such a plenoptic camera considersit as a cascade of a main lens system followed by a microlens system.The basic operation of the cascade system is as follows. Rays focused bythe main lens 104 are separated by the microlenses 106 and captured onthe sensor 108. At their point of intersection, the rays have the sameposition but different slopes. This difference in slopes causes theseparation of the rays when they pass through a microlens-space system.In more detail, each microlens functions to swap the positional andangular coordinates of the radiance, then this new positionalinformation is captured by the sensor 108. Because of the swap, itrepresents the angular information at the microlens. As a result, eachmicrolens image captured by sensor 108 represents the angularinformation for the radiance at the position of the optical axis of thecorresponding microlens.

The light-field is the radiance density function describing the flow ofenergy along all rays in three-dimensional (3D) space. Since thedescription of a ray's position and orientation requires four parameters(e.g., two-dimensional positional information and two-dimensionalangular information), the radiance is a four-dimensional (4D) function.This function may be referred to as the plenoptic function. Image sensortechnology, on the other hand, is only two-dimensional, and light-fieldimagery must therefore be captured and represented in flat (twodimensional) form. A variety of techniques have been developed totransform and capture the 4D radiance in a manner compatible with 2Dsensor technology. This may be referred to as a flat representation ofthe 4D radiance (or light-field), or simply as a flat.

To accommodate the extra degrees of dimensionality, extremely highsensor resolution is required to capture flat radiance. Even so, imagesare rendered from a flat at a much lower resolution than that of thesensor, i.e., at the resolution of the radiance's positionalcoordinates. If the angular information is finely sampled, then a largenumber of pixels from the flat are being used to create just one pixelin the rendered image. Each microlens determines only one pixel in therendered image; when the angular information under one microlens isintegrated, only one pixel is determined in the rendered image. Therendered image may thus have a resolution that is orders of magnitudesmaller than the raw flat itself. For example, in an exampleconventional light-field camera, a flat may be represented in 2D with a24,862×21,818 pixel array. The 4D radiance that is represented may, forexample, be 408×358×61×61. With existing rendering techniques, imagesare rendered from this radiance at 408×358, i.e., 0.146 megapixel. Notonly is this a disappointingly modest resolution (any cell phone todaywill have better resolution), any particular rendered view only uses oneout of every 3,720 pixels from the flat imagery. The large disparitybetween the resolution of the flat and the rendered images isextraordinarily wasteful for photographers who are ultimately interestedin taking photographs rather than capturing flat representations of theradiance.

SUMMARY

Various embodiments of a method and apparatus for full-resolutionlight-field capture and rendering are described. Embodiments of afull-resolution radiance camera, and of a method for renderinghigh-resolution images from flat 2D representations of the 4Dlightfield, referred to herein as flats, captured by embodiments of thefull-resolution radiance camera, are described. Images rendered fromflats using conventional light-field cameras and conventionallight-field rendering methods are at significantly low resolutions.Embodiments of the full-resolution radiance camera and of thefull-resolution light-field rendering method more adequately meet theresolution and image size expectations of modern photography than doconventional light-field cameras and rendering methods. The term “fullresolution” does not directly refer to sensor resolution of the camera,but instead refers to resolution as supported by the captured radiancedata.

In embodiments of the radiance camera, the microlenses are focused on animage created by the main lens (the image plane of the main lens) withinthe camera, instead of being focused on the main lens itself, as inconventional plenoptic cameras. This serves to increase or maximizespatial resolution, and to thus achieve sharper, higher spatialresolution microlens images. In the image plane, there is a real imageof a scene in front of the camera and refracted by the main lens to theimage plane, but there is nothing there physically (other than light);the image plane is simply a plane location in space that can beconsidered to have an image “in the air” as created by the main lens.The microlenses, being focused on the image plane instead of on the mainlens, can capture the image of the scene at the image plane. Eachmicrolens captures a small area or region of the image at the imageplane and maps or projects the captured region onto a correspondingregion of the photosensor. The imaging property of the radiance cameramay be viewed as two steps; from the world through the main lens to theimage plane, and then from the image plane through the microlenses tothe photosensor. This is similar to a cascade of two cameras, but thesecond camera is actually many small cameras, as each microlens iseffectively a little camera that captures a small image from the imageplane. In further contrast to conventional plenoptic cameras, themicrolenses in embodiments of the radiance camera may be located at, ormay be moved to, distances greater than f or less than f from thephotosensor, where f is the focal length of the microlenses. In anexample embodiment, the array of microlenses may be placed at distance4/3 f from the photosensor. Other embodiments may place the array ofmicrolenses at other distances, greater than or less than f, from thephotosensor. In addition, embodiments of radiance cameras in which thedistance of the microlens array from the photosensor is variable oradjustable, and in which other characteristics of the camera may beadjustable, are described. Various embodiments of the full-resolutionradiance camera implemented in digital cameras and in film cameras areanticipated, and example embodiments of both types are described.

In an embodiment of a full-resolution light-field rendering method, aflat captured by a radiance camera may be obtained. Microimages in areasof the flat may be examined to determine the direction of movement ofedges in the microimages relative to a direction of movement. If it isdetermined that edges in microimages of an area are moving relative tothe microimage centers in the same direction as the direction ofmovement, the microimages in that area may be inverted relative to theirindividual centers. In some embodiments, examination of the microimagesto determine the direction of movement of edges may be performed by auser via a user interface. The user may mark or otherwise indicate areasthat the user determines need be inverted via the user interface. Insome embodiments, examination of the microimages to determine thedirection of movement of edges may be performed automatically insoftware. The microimages may each be cropped to produce an m₁×m₂subregion or crop of each microimage, where at least one of m₁ and m₂ isan integer greater than two. The subregions or crops from themicroimages may then be appropriately assembled to produce a finalhigh-resolution image of the scene.

In some embodiments, two or more images rendered from a flat may becombined to produce a higher-quality output image. For example, in someembodiments, the microimages in a flat may all be inverted, and theinverted microimages appropriately assembled to produce a firstintermediate image. A second intermediate image may be generated withoutinverting the microimages prior to assembling. The two intermediateimages may then be combined to produce a higher-quality output image.The combination of the two images may be performed manually by a uservia a user interface, or may be performed automatically in software.

By focusing the microlenses on the image produced by the main lens,embodiments of the radiance camera are able to fully capture thepositional information of the light-field. Embodiments of thefull-resolution light-field rendering method may be used to renderfull-resolution images from flats captured by embodiments of theradiance camera, producing output images at a dramatically higherresolution than conventional light-field rendering techniques.Embodiments may render images at spatial resolutions that meet theexpectations of modern photography (e.g., 10 megapixel and beyond),making light-field photography much more practical.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates imaging in a conventional camera.

FIG. 2 illustrates an example prior art light-field camera, or cameraarray, which employs an array of two or more objective lenses.

FIG. 3 illustrates an example prior art plenoptic camera that employs asingle objective lens and a microlens array.

FIG. 4 further illustrates an example prior art plenoptic camera.

FIG. 5A shows a raw light-field image, or flat, as captured by aplenoptic camera.

FIG. 5B shows a final image rendered from the flat of FIG. 5A accordingto a conventional rendering method.

FIG. 5C shows a final image rendered from the flat of FIG. 5A accordingto an embodiment of the full-resolution light-field rendering method.

FIG. 6 is a block diagram illustrating a full-resolution radiance cameraaccording to one embodiment.

FIG. 7 illustrates an example embodiment of a radiance camera withvarious other elements that may be integrated in the camera.

FIG. 8 illustrates an example embodiment of a radiance camera 200 basedon a large-format film camera.

FIG. 9 shows an example crop from a flat acquired with a plenopticcamera.

FIG. 10 illustrates the telescopic case for a plenoptic camera.

FIG. 11 shows a crop from the roof area in FIG. 9, and visuallyillustrates the “telescopic” behavior in light-field cameras.

FIG. 12 illustrates the binocular case for a plenoptic camera.

FIG. 13 shows a crop from the tree area in FIG. 9, and visuallyillustrates the “binocular” behavior in light-field cameras.

FIG. 14A shows the ray geometry in the telescopic case for n=4.

FIG. 14B shows the ray geometry in the telescopic case for n=2.

FIG. 15 illustrates a lens circle (or microimage) of diameter D and apatch or crop of size m₁×m₂, where at least one of m₁ and m₂ is aninteger greater than or equal to 2.

FIG. 16 shows a zoom into an example microlens array.

FIG. 17 shows a portion of a digitized flat.

FIGS. 18A through 18C show output images rendered from a flat usingconventional rendering methods.

FIG. 19 show a full-resolution rendering of a light-field, renderedassuming the telescopic case according to one embodiment of thefull-resolution light-field rendering method.

FIG. 20 shows a full-resolution rendering of a light-field, renderedassuming the binocular case according to one embodiment of thefull-resolution light-field rendering method.

FIG. 21 is a flow chart illustrating how light is directed within aradiance camera according to one embodiment.

FIG. 22 is a flowchart of a full-resolution light-field rendering methodaccording to one embodiment.

FIG. 23 illustrates a full-resolution light-field rendering modulerendering a high-resolution image from a flat captured, for example, bya radiance camera, according to one embodiment.

FIG. 24 illustrates an example computer system that may be used inembodiments.

FIG. 25 is a flowchart of a full-resolution light-field rendering methodin which multiple images are rendered from a flat and combined toproduce a final high-resolution output image, according to someembodiments.

FIG. 26 shows an example full-resolution rendering of a light-field inwhich foreground and background portions of the images shown in FIGS. 19and 20 have been combined, according to one embodiment of thefull-resolution light-field rendering method.

While the invention is described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the invention is not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit the invention tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention. The headings used herein arefor organizational purposes only and are not meant to be used to limitthe scope of the description. As used throughout this application, theword “may” is used in a permissive sense (i.e., meaning having thepotential to), rather than the mandatory sense (i.e., meaning must).Similarly, the words “include”, “including”, and “includes” meanincluding, but not limited to.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following detailed description, numerous specific details are setforth to provide a thorough understanding of claimed subject matter.However, it will be understood by those skilled in the art that claimedsubject matter may be practiced without these specific details. In otherinstances, methods, apparatuses or systems that would be known by one ofordinary skill have not been described in detail so as not to obscureclaimed subject matter.

Some portions of the detailed description which follow are presented interms of algorithms or symbolic representations of operations on binarydigital signals stored within a memory of a specific apparatus orspecial purpose computing device or platform. In the context of thisparticular specification, the term specific apparatus or the likeincludes a general purpose computer once it is programmed to performparticular functions pursuant to instructions from program software.Algorithmic descriptions or symbolic representations are examples oftechniques used by those of ordinary skill in the signal processing orrelated arts to convey the substance of their work to others skilled inthe art. An algorithm is here, and is generally, considered to be aself-consistent sequence of operations or similar signal processingleading to a desired result. In this context, operations or processinginvolve physical manipulation of physical quantities. Typically,although not necessarily, such quantities may take the form ofelectrical or magnetic signals capable of being stored, transferred,combined, compared or otherwise manipulated. It has proven convenient attimes, principally for reasons of common usage, to refer to such signalsas bits, data, values, elements, symbols, characters, terms, numbers,numerals or the like. It should be understood, however, that all ofthese or similar terms are to be associated with appropriate physicalquantities and are merely convenient labels. Unless specifically statedotherwise, as apparent from the following discussion, it is appreciatedthat throughout this specification discussions utilizing terms such as“processing,” “computing,” “calculating,” “determining” or the likerefer to actions or processes of a specific apparatus, such as a specialpurpose computer or a similar special purpose electronic computingdevice. In the context of this specification, therefore, a specialpurpose computer or a similar special purpose electronic computingdevice is capable of manipulating or transforming signals, typicallyrepresented as physical electronic or magnetic quantities withinmemories, registers, or other information storage devices, transmissiondevices, or display devices of the special purpose computer or similarspecial purpose electronic computing device.

Various embodiments of a method and apparatus for full-resolutionlight-field capture and rendering are described. Embodiments of afull-resolution radiance camera, and of a method for renderinghigh-resolution images from flat 2D representations of the 4Dlightfield, referred to herein as flats, captured by embodiments of thefull-resolution radiance camera, are described. The method for renderinghigh-resolution images from flats captured by embodiments of thefull-resolution radiance camera may be referred to as a full-resolutionlight-field rendering method, or simply as the light-field renderingmethod. The term “full resolution” does not directly refer to sensorresolution of the camera, but instead refers to resolution as supportedby the captured radiance data. For simplicity, the full-resolutionradiance camera may be referred to as a radiance camera.

Light-field photography enables many new possibilities for digitalimaging because it captures both spatial and angular information, i.e.,the full four-dimensional radiance, of a scene. High-resolution isrequired in order to capture four-dimensional data with atwo-dimensional sensor. However, images rendered from this data asprojections of the four-dimensional radiance onto two spatial dimensionsusing conventional light-field cameras and conventional light-fieldrendering methods are at significantly lower resolutions. Embodiments ofthe full-resolution radiance camera and of the full-resolutionlight-field rendering method more adequately meet the resolution andimage size expectations of modern photography than do conventionallight-field cameras and rendering methods.

In embodiments of the full-resolution radiance camera, the microlensesin the microlens array are focused on the image plane of the main cameralens, rather than on the main camera lens itself as in conventionalplenoptic cameras. In the image plane, there is a real image of a scenein front of the camera and refracted by the main lens to the imageplane, but there is nothing there physically (other than light); theimage plane is simply a plane location in space that can be consideredto have an image “in the air” as created by the main lens. Themicrolenses, being focused on the image plane instead of on the mainlens, can capture the image of the scene at the image plane. Eachmicrolens captures a small area or region of the image at the imageplane and maps or projects the captured region onto a correspondingregion of the photosensor. The imaging property of the radiance cameramay be viewed as two steps: from the world through the main lens to theimage plane, and then from the image plane through the microlenses tothe photosensor. This is similar to a cascade of two cameras, but thesecond camera is actually many small cameras, as each microlens iseffectively a little camera that captures a small image from the imageplane. This is also similar to the way a telescope operates. By focusingthe microlenses on the image produced by the main lens, embodiments ofthe radiance camera are able to fully capture the positional informationof the radiance. Embodiments of the full-resolution light-fieldrendering method may be used to render full-resolution images from flatscaptured by embodiments of the radiance camera, producing output imagesat a dramatically higher resolution than conventional light-fieldrendering techniques. Embodiments may render images at spatialresolutions that meet the expectations of modern photography (e.g., 10megapixel and beyond), making light-field photography much morepractical.

An analysis of light-field camera structure and optics is given belowthat provides insight on the interactions between the main lens systemand the microlens array in light-field cameras. Based on results of thisanalysis, embodiments exploit the fact that, at every plane of depth,the radiance contains a considerable amount of positional informationabout the scene, encoded in the angular information at that plane.Accordingly, embodiments may be referred to as full-resolution becauseembodiments make full use of both angular and positional informationthat is available in the four-dimensional radiance, as shown in theanalysis. In contrast to super-resolution techniques, which createhigh-resolution images from sub-pixel shifted low-resolution images,embodiments render high-resolution images directly from the radiancedata. Moreover, embodiments may generate light-field images that areamenable to radiance processing techniques such as Fourier slicerefocusing.

FIGS. 5A through 5C show, for comparison, results from a conventionalplenoptic camera and rendering method and results from exampleembodiments of a full-resolution radiance camera and full-resolutionlight-field rendering method as described herein. FIG. 5A shows a rawlight-field image as captured by a plenoptic camera. Note that, to theuntrained human eye, the raw light-field image captured by aconventional plenoptic camera may look similar to the raw light-fieldimage captured by an embodiment of the full-resolution radiance camera.FIG. 5B shows a conventionally rendered final image, and FIG. 5C shows afinal image rendered according to an embodiment of the full-resolutionlight-field rendering method as described herein. Even in this small,grayscale format, a drastic improvement in spatial resolution in FIG. 5Cwhen compared to the spatial resolution in FIG. 5B is easily observable.

Full-Resolution Radiance Cameras

Various embodiments of a full-resolution radiance camera are described.In conventional plenoptic cameras such as those illustrated in FIGS. 3and 4, the microlenses are placed and adjusted accurately to be exactlyat one focal length f from the photosensor, where f is the focal lengthof the microlenses. In addition, in conventional plenoptic cameras, themicrolens array is fixed at the image plane of the main or objectivelens of the camera, and the microlenses in the array are focused atinfinity. In contrast, in embodiments of the radiance camera describedherein, in order to increase or maximize spatial resolution, i.e., toachieve sharper, higher spatial resolution, microlens images, themicrolenses are focused on the image created by the main lens inside thecamera and in front of the microlenses (the image plane of the mainlens), instead of being focused on the main lens itself, as inconventional plenoptic cameras. In further contrast to conventionalplenoptic cameras, the microlenses in embodiments of the radiance cameradescribed herein may be located at, or may be moved to, distancesgreater than f or less than f from the photosensor, where f is the focallength of the microlenses. In one embodiment, the array of microlensesmay be placed at distance 4/3 f from the photosensor. Other embodimentsmay place the array of microlenses at other distances that are multiplesoff e.g. 1.5 f or ¾ f. In addition, embodiments of radiance cameras inwhich the distance of the microlens array from the photosensor isvariable or adjustable, and in which other characteristics of the cameramay be adjustable, are described. For example, in one embodiment, thedistance of the microlens array from the photosensor may be adjustablewithin the range 0.5 f to 1.5 f. For the telescopic case (the distanceof the microlens array from the photosensor >f), a maximum usefuldistance may be 1.5 f, although distances greater than 1.5 f may bepossible, if not practical. Thus, for the telescopic case, a practicalrange for the distance of the microlens array from the photosensor maybe f<b≦1.5 f.

Various embodiments of the full-resolution radiance camera implementedin digital cameras and in film cameras are anticipated, and exampleembodiments of both types are described. In digital cameras, thephotosensor is a digital light-capturing device or medium such as acharge-coupled device (CCD) that captures and records the light indigital format. In film cameras, the photosensor is a film. Thus,“photosensor” as used herein refers to digital media that are used indigital cameras to capture light and to film media that are used in filmcameras to capture light, and more generally to any device or mediumthat may be used to capture light. Light-field images captured on a filmusing film camera embodiments may subsequently be digitized, for exampleusing a high-resolution scanner, so that the captured light-field may berendered, for example using the full-resolution light-field renderingmethod described herein, to produce high-resolution output images.Light-field images captured using digital camera embodiments may bedirectly rendered.

In addition to digital and film embodiments, fixed and adjustableembodiments of both digital camera and film camera embodiments of thefull-resolution radiance camera are anticipated, and example embodimentsof both types are described. In a fixed embodiment, the photosensor andthe microlens array are at a fixed distance b from each other (thedistance b is a multiple off for example 4/3 f ¾ f, or 1.5 f, where f isthe focal length of the microlenses). Note that b is used herein todesignate the distance between the microlenses and the photosensor,while a is used herein to designate the distance between the microlensesand the image plane of the main or objective lens. In some embodiments,the microlens array/photosensor combination may be fixed at a locationin the camera body. In some embodiments, the microlens array may befixed in optical characteristics as well as in its physical location. Insome embodiments, the main lens of the camera may also be fixed inoptical characteristics and location, while possibly allowing forchanges in shutter speed, aperture, focusing, etc. In adjustableembodiments, various manual or automatic mechanisms may be employed tochange the distance b between the photosensor and the microlens array,to change the location of the microlens array/photosensor combination inthe camera body, to change the distance from the main lens to themicrolens array, to change the distance a between the microlenses andthe image plane, and/or to swap or replace various components such asthe microlens array and the main lens. In addition, the main lens of thecamera may be swappable to use different main lenses, and may beadjustable according to aperture, shutter speed, focusing, distance fromthe microlens array, and so on. Embodiments where the microlens arraymay be swappable, so that microlens arrays with different numbers ofmicrolenses and/or microlenses with different optical characteristicsmay be used, are also possible.

The optical characteristics of the optical system, including the opticalcharacteristics of the lenses and the distances between the variouscomponents or elements, is important in capturing light-fields that maybe rendered to yield high-resolution output images as described herein.Thus, in fixed embodiments, the microlenses, main lens, photosensor, andthe relative physical location of these components in the camera may bedetermined according to the formulas and equations described herein tocapture appropriate and satisfactory light-field images. In adjustableembodiments, some embodiments may include automated mechanisms thatautomatically adjust the positioning or other aspects of one or more ofthe components to capture appropriate and satisfactory light-fieldimages. For example, if the user adjusts or replaces one component, thecamera may automatically adjust one or more other components tocompensate for the change. Alternatively, a human operator of anadjustable radiance camera may manually adjust the positioning or otheraspects of one or more of the components, may replace one or morecomponents with units that have different characteristics, or may insertother components (e.g., microsheet glass, as described below) to captureappropriate and satisfactory light-field images.

FIGS. 6 through 8 illustrate example film camera and digital cameraembodiments of a radiance camera as described herein, and furtherillustrate both fixed and adjustable embodiments of the radiance camera.It is noted that these are example embodiments, and are not intended tobe limiting. Other embodiments are possible and anticipated.

FIG. 6 is a block diagram illustrating a full-resolution radiance cameraaccording to one embodiment. Radiance camera 200 may include a main(objective) lens 230, a microlens array 220, and a photosensor 210.Microlens array 220 may be located at a distance greater than f fromphotosensor 210, where f is the focal length of the microlenses in array220. In addition, the microlenses in array 220 are focused on the imageplane 240 of the main lens 230. In contrast, in conventional plenopticcameras such as plenoptic camera 102 of FIGS. 3 and 4, the microlensarray 106 is fixed at distance f from photosensor 108, and themicrolenses in array 106 are focused on the main lens 104. In someembodiment, photosensor 210 may be conventional film; in otherembodiments, photosensor 210 may be a device for digitally capturinglight, for example a CCD. In one embodiment of a microlens array 220that may be used in embodiments of radiance camera 200, or in otherembodiments as illustrated in FIGS. 7 and 8, the microlens array 220 mayinclude 146,000 microlenses of diameter 0.25 mm and focal length 0.7 mm.Other configurations of microlens array 220, including different numbersof microlenses and/or microlenses with different opticalcharacteristics, are possible and anticipated. FIG. 16 shows a zoom intoan example microlens array, and shows individual microlenses and (black)chromium mask between the microlenses.

FIG. 7 illustrates an example embodiment of radiance camera 200 withvarious other elements that may be integrated in the camera 200. In someembodiments of radiance camera 200, the objective lens 230, themicrolens array 220, and the photosensor 210 may be fixed. In otherembodiments, one or more of the above elements may be replaceable and/oradjustable. In some embodiment, photosensor 210 may be conventionalfilm; in other embodiments, photosensor 210 may be a device fordigitally capturing light, for example a CCD. In general, embodiments ofa radiance camera 200 as described herein may include, in addition tomain lens 230, microlens array 220, and photosensor 210, any other typeof elements and features commonly found in digital cameras or othercameras including light-field and plenoptic cameras and large-formatfilm cameras, and may also include additional elements and features notgenerally found in conventional cameras.

In one embodiment, a full-resolution light-field rendering method forrendering high-resolution images from light-fields captured by radiancecamera 200, and/or other image processing algorithms for application tolight-fields captured by embodiments of radiance camera 200, may beimplemented in captured data processing module 260. Captured dataprocessing module 260 may be implemented in hardware, software, or acombination thereof. Alternatively, light-fields captured by radiancecamera 200 may be rendered according to the full-resolution light-fieldrendering method implemented in a full-resolution light-field renderingmodule executing on a separate device, e.g. a computer system, togenerate one or more high-resolution output images of a captured scene,as described herein. An example computer system in which embodiments ofthe full-resolution light-field rendering method may be implemented isillustrated in FIG. 24.

A radiance camera 200 may include a shutter 314. Shutter 314 may belocated in front of or behind objective lens 230. A radiance camera 200may include one or more processors 300. A radiance camera 200 mayinclude a power supply or power source 304, such as one or morereplaceable or rechargeable batteries. A radiance camera 200 may includea memory storage device or system 302 for storing captured light-fieldimages and/or rendered final images or other information such assoftware. In one embodiment, the memory 302 may be a removable/swappablestorage device such as a memory stick. A radiance camera 200 may includea screen 306 (e.g., an LCD screen) for viewing scenes in front of thecamera prior to capture and/or for viewing previously captured and/orrendered images. The screen 306 may also be used to display one or moremenus or other information to the user. A radiance camera 200 mayinclude one or more I/O interfaces 312, such as FireWire or UniversalSerial Bus (USB) interfaces, for transferring information, e.g. capturedlight-field images, software updates, and so on, to and from externaldevices such as computer systems or even other cameras. A radiancecamera 200 may include a shutter release 308 that is activated tocapture a light-field image of a subject or scene.

A radiance camera 200 may include one or more controls 310, for examplecontrols for controlling optical aspects of the radiance camera 200 suchas shutter speed, one or more controls for viewing and otherwisemanaging and manipulating captured images stored in a memory on thecamera, etc. An adjustable radiance camera 200 may include one or morecontrols for adjusting the relative location of (the distance between)the components in the camera 200, such as the distance b betweenmicrolens array 220 and photosensor 210. An adjustable radiance camera200 may include one or more manual or automatic adjusting mechanism(s)320, or adjusters, configured to adjust the relative location of (thedistance between) the components in the camera 200, such as the distanceb between microlens array 220 and photosensor 210. In some embodiments,the adjusting mechanisms 320 may act to adjust one or more componentsresponsively to controls 310.

FIG. 8 illustrates an example embodiment of a radiance camera 200 basedon a large-format film camera. In conjunction with currenthigh-resolution scanners used to digitize captured images from negativesor prints, large-format film camera embodiments are capable of up to 1gigapixel, or even higher, resolution for the flat (a flat is a 2Drepresentation of the 4D radiance). An example embodiment may, forexample, be implemented in large-format film camera using a 135 mmobjective lens 430 and 4×5 format film as the “photosensor” (inlarge-format cameras, single negatives of film are generally placed in afilm holder 402 or cartridge that can be inserted into and removed fromthe camera body). Other objective lenses and/or other film formats, forexample 8×10 format film, may be used in various embodiments. Radiancecamera 400 includes a microlens array 406. FIG. 16 shows a zoom into anexample microlens array, and shows individual microlenses and (black)chromium mask between the microlenses. In one embodiment of a microlensarray that may be used in embodiments of radiance camera 400, or inother embodiments as illustrated in FIGS. 6 and 7, the microlens array406 may include 146,000 microlenses of diameter 0.25 mm and focal length0.7 mm. Other configurations of microlens array 406, including differentnumbers of microlenses and/or microlenses with different opticalcharacteristics, are possible and anticipated.

In one embodiment, a mechanism inside a film holder 402 of thelarge-format film camera holds the microlens array 406 so that the flatside of the glass base of the array 406 is pressed against the film. Inone embodiment, the thickness of the microlens array 406 is such that,when placed against the film, the microlenses are distance f from thefilm. Other configurations of microlens arrays 406 are possible, and theconfiguration of the large-format film camera makes it possible toeasily change configurations of microlenses by simply using a differentmicrolens array 406. Microsheets 404 of glass may be used in theassembly as spacers or shims between the microlens array 406 and thefilm in film holder 402 to increase the distance from the microlensesand the film to be greater than f (e.g., 4/3 f). An example thickness ofa microsheet 404 that may be used is 0.23 mm. Inserting microsheet glass404 provides spacing in a rigorously controlled manner. In oneembodiment, additional spacing may be created by adding a singlemicrosheet 404 between the film holder 402 and the microlens array 406in order to displace the microlenses by an additional ⅓ f, approximately0.2 mm from the sensor. Additional microsheets 404 may be added toprovide additional spacing. In some embodiments, other mechanisms thanmicrosheet glass may be used as spacers between the microlens array 406and film holder 402 to adjust the distance between the microlens array406 and film holder 402.

As illustrated in FIG. 8, in one embodiment, the film holder 402 andmicrolens array 406 may be coupled to create assembly 410. One or moremicrosheets 404 may optionally be inserted between the film holder 402and microlens array 406 to provide additional spacing as necessary ordesired. The assembly 410 may then be inserted into the large-formatfilm camera. The combination of the large-format film camera and theassembly 410 effectively forms a radiance camera 400. Radiance camera400 may then be used to capture a flat of a scene on the film in filmholder 402. A flat is a 2D representation of the 4D lightfield. Theassembly 410 may then be removed from the camera 400, disassembled, andthe film may be appropriately processed. The film negative and/or aprint of the flat may then be digitized, for example using ahigh-resolution scanner or a device that generates digital images fromnegatives. The digitized flat may be stored to a storage device, such asa disk drive, DVD, CD, etc. The digitized flat may be rendered accordingto the full-resolution light-field rendering method, implemented in afull-resolution light-field rendering module executing on a computersystem, to generate one or more high-resolution output images of thescene as described herein. An example computer system in whichembodiments of the full-resolution light-field rendering method may beimplemented is illustrated in FIG. 24.

An analysis of the full-resolution light-field rendering methods andapparatus provided herein shows that focusing the microlenses on theimage plane of the main lens in the radiance camera, rather thanfocusing on the main lens itself as in conventional plenoptic cameras,enables embodiments of the full-resolution light-field rendering methodsand apparatus to more fully exploit positional information available inthe captured flat (i.e., the 2D representation of the 4D light-field)captured by the light-field camera). Based on good focusing andhigh-resolution of the microlens images, embodiments of the describedmethods and apparatus are able to achieve very high-resolution ofrendered images when compared to conventional plenoptic cameras andconventional rendering methods. For example, one embodiment achieves a27× increase in resolution in each spatial dimension when compared toresults from conventional plenoptic cameras and conventional renderingmethods.

Full-Resolution Light-Field Rendering Method

Embodiments of a method and apparatus for rendering high-resolutionimages from a light-field, for example captured by embodiments of thefull-resolution radiance camera, are described. The method for renderinghigh-resolution images from the light-field may be referred to as afull-resolution light-field rendering method. The light-field renderingmethod may be referred to as full-resolution because the method makesfull use of both positional and angular information available in thecaptured radiance data. The full-resolution light-field rendering methodmay be implemented as or in a tool, module, library function, plug-in,stand-alone application, etc. For simplicity, implementations ofembodiments of the full-resolution light-field rendering method mayreferred to as a full-resolution light-field rendering module.Alternatively, or in addition, other light-field rendering or processingtechniques may be applied to captured flats by a full-resolutionlight-field rendering module, and/or by other modules. FIG. 24illustrates an example computer system on which embodiments of afull-resolution light-field rendering module may be implemented.

A description of the full-resolution light-field rendering method and ananalysis of the limits and tradeoffs of the method are presented. Theeffectiveness of the full-resolution light-field rendering method whencompared to conventional methods may be demonstrated experimentally byrendering images from a 542-megapixel light-field using a conventionalrendering approach and using the full-resolution light-field renderingmethod described herein. In the experiments, the conventional renderingmethods produce a 0.146-megapixel final image, while the full-resolutionlight-field rendering method produces a 106-megapixel final image.Experimental results show that our method may produce full-resolutionimages that approach the resolution that would have been captureddirectly with a conventional (non-light-field) high-resolution camera.

Plenoptic Camera Modes of Behavior

The full-resolution light-field rendering method may be derived byanalyzing the optical system of the plenoptic camera. First, someobservations of captured flats, which are 2D representations of the 4Dlight-field, are presented, and these observations are used to motivatethe subsequent analysis.

FIG. 9 shows an example crop from a raw flat acquired with a plenopticcamera. In FIG. 9, repeated edges inside multiple circles may beobserved. Each microlens in the microlens array creates a microimage;the resulting flat is thus an array of microimages. On a large scale,the overall image may be perceived, whereas the correspondence betweenthe individual microlens images and the large scale scene is lessobvious. Interestingly, as will be shown, it is thisrelationship—between what is captured by the microlenses and what is inthe overall scene—that may be exploited in embodiments to createhigh-resolution images.

In FIG. 9, on a small scale, a number of clearly distinguishablefeatures inside the circles, such as edges, may be observed. Edges areoften repeated from one circle to the next. The same edge (or feature)may be seen in multiple circles, in a slightly different position thatshifts from circle to circle. If the main camera lens is manuallyrefocused, a given edge can be made to move and, in fact, change itsmultiplicity across a different number of consecutive circles.

Repetition of features across microlenses is an indication that thatpart of the scene is out of focus. When an object from the large-scalescene is in focus, the same feature appears only once in the array ofmicroimages.

In interpreting the microimages, it is important to note that, as withthe basic conventional camera described above, the operation of a basicplenoptic camera is far richer than a simple mapping of the radiancefunction at some plane in front of the main lens onto the sensor. Thatis, there are an essentially infinite number of mappings from the scenein front of the lens onto the image sensor. For one particular distance,this corresponds to a mapping of the radiance function. What thecorrespondence is for parts of the scene at other distances—as well ashow they manifest themselves at the sensor—is less obvious. This will bethe topic of the remaining part of this section.

Next, two limiting cases are considered which can be recognized in thebehavior of the plenoptic camera: Telescopic (where the distance betweenthe photosensor and the microlens array, b, is greater than the focallength f of the microlenses in the array) and Binocular (where b is lessthan f). Neither of those cases is exact for a true plenoptic camera,but their fingerprints can be seen in every plenoptic image. As will beshow, both are achievable, and are very useful.

Plenoptic Camera: Telescopic Case

FIG. 10 illustrates the telescopic case (b>f) for a plenoptic camera. Aplenoptic camera may be considered as an array of (Keplerian) telescopeswith a common objective lens. (For the moment the issue of microlensesnot being exactly focused for that purpose will be ignored.) Eachindividual telescope in the array has a microcamera (an eyepiece lensand the eye) inside the big camera. Just like any other camera, thismicrocamera is focused onto one single plane, and maps the image fromthe plane onto the retina, inverted and reduced in size. A camera can befocused only for planes at distances ranging from f to infinity (∞)according to the lens equation:

${\frac{1}{a} + \frac{1}{b}} = \frac{1}{f}$

Here, a, b, and f have the same meaning as for the big camera, except ona smaller scale. It can be seen that since a and b must be positive, itis not possible to focus closer than f. In a conventional plenopticcamera, the image plane is fixed at the microlenses. It may be morenatural to consider the image plane fixed at distance f in front of themicrolenses. In both cases, microimages are out of focus.

Following the movement of an edge from circle to circle, characteristicbehavior of telescopic imaging in the flat may be observed. FIG. 11shows a crop from the roof area in FIG. 9. FIG. 11 may be used tovisually illustrate the “telescopic” behavior. It is possible to observein FIG. 11 that the edge is repeated two times when moving away from theroof. The farther from the roof a circle is, the farther the edgeappears inside that circle. Moving in any given direction, the edgemoves relative to the circle centers in the same direction. Oncedetected in a given area, this behavior is consistent (valid in alldirections in that area). Careful observation shows that images in thesmall circles are indeed inverted patches from the high-resolutionimage, as if observed through a telescope.

For the telescopic case, a practical range for b may be f<b≦1.5 f.

Plenoptic Camera: Binocular Case

FIG. 12 illustrates the binocular (Galilean type telescope) case (b<f)for a plenoptic camera. FIG. 13 shows a crop from the tree area in FIG.9, and is used to illustrate details of “binocular” imaging inlight-field cameras. Note that the image is not inverted in FIG. 13. Aplenoptic camera may also be considered as an “incompletely focused”camera, i.e., a camera focused behind the film plane (as in a Galileantelescope/binoculars). If an appropriate positive lens is placed infront of the film, the image would be focused on the film. For aGalilean telescope, this is the lens of the eye that focuses the imageonto the retina. For a plenoptic camera, this role is played by themicrolenses with focal length f. In the binocular case, the microlenseswould need to be placed at a distance smaller than f from the film. Notealso that while the telescopic operation inverts the inside image, thebinocular operation does not invert it.

As with telescopic imaging, characteristic behavior of binocular imagingcan be observed in the plenoptic camera. See FIG. 13, which is a cropfrom the top left corner in FIG. 9. In FIG. 13, it can be observed thatedges are repeated about two or three times when moving away from thebranch. The farther from the branch, the closer to the branch the edgeappears inside the circle. Moving in any given direction, the edge movesrelative to the circle centers in the opposite direction. Once detectedin a given area, this behavior is consistent (valid in all directions inthat area). This is due to the depth in the image at that location.Careful observation shows that images in the small circles are in factpatches from the corresponding area in the high-resolution image, onlyreduced in size. The more times the feature is repeated in the circles,the smaller it appears and thus a bigger area is imaged inside eachindividual circle.

To summarize, an approximately focused plenoptic camera (i.e., aplenoptic camera where b≠f) may be considered as an array ofmicrocameras looking at an image plane in front of the array or behindthe array. Each microcamera images only a small part of that plane. Theshift between those small images is obvious from the geometry, asexplained below in the section titled Analysis. If at least onemicrocamera could image this entire plane, it could directly capture ahigh-resolution image. However, the small images are limited in size bythe main lens aperture.

The magnification of these microcamera images, and the shift betweenthem, is defined by the distance to the image plane. The distance can beat positive or negative distance from the microlenses, corresponding tothe telescopic (positive) and binocular (negative) cases describedabove. By slightly adjusting the plane of the microlenses (so that thelenses are in focus), embodiments can make use of the telescopic orbinocular behavior to generate a high-resolution image from the flat.This process is described in the following sections.

Analysis

In some embodiment, microlenses may not be focused exactly on the planethat is to be imaged, causing the individual microlens images to beblurry. This may limit the amount of resolution that can be achieved.One way to improve such results would be deconvolution. Another waywould be to stop down the microlens apertures.

In FIGS. 14A and 14B, the case of a “plenoptic” camera using a pinholearray instead of microlens array is considered. In FIGS. 14A and 14B, anarray of pinholes (or microlenses) maps the image in front of the arrayto the sensor. The distance to the image defines the magnificationfactor M=n−1. In ray optics, in theory, pinhole images produce nodefocus blur, and in this way are perfect. But this is in theory; in thereal world, pinholes are replaced with finite but small apertures andmicrolenses.

From the lens equation:

${\frac{1}{a} + \frac{1}{b}} = \frac{1}{f}$

it can be seen that, if the distance to the object is a=nf, the distanceto the image would be:

$b = \frac{nf}{n - 1}$ $n = \frac{b}{b - f}$

The geometric magnification factor may be defined as M=a/b, which bysubstitution gives:

M=n−1

FIG. 14A shows the ray geometry in the telescopic case for n=4, and FIG.14B shows the ray geometry in the telescopic case for n=2. Note that thedistance b from the microlenses to the sensor is always greater than f(this is not represented in FIGS. 14A and 14B). Looking at the geometryin FIGS. 14A and 14B, the images are M times smaller, inverted, andrepeated M times.

Full-Resolution Light-Field Rendering Algorithm

Two distinct behaviors (telescopic and binocular) are described above,and embodiments of the full-resolution light-field rendering method mayexecute a different action based on which behavior is observed in themicroimages contained in the flat captured by a radiance camera. In oneembodiment, if the full-resolution light-field rendering method detectsedges (or features) moving relative to the microimage centers (themicroimages are generally circular, so may be referred to as circles) inthe same direction as the direction of movement, all microimages in thatarea are inverted relative to their individual centers (this is thetelescopic case). If the full-resolution light-field rendering methoddetects edges moving relative to the microimage centers in a directionopposite to the direction of movement, the method does nothing (this isthe binocular case). In some embodiments, examination of the microimagesto determine the direction of movement of edges may be performed by auser via a user interface. The user may mark or otherwise indicate areasthat the user determines need be inverted via the user interface. Insome embodiments, examination of the microimages to determine thedirection of movement of edges may be performed automatically insoftware.

The small circles, or microimages, in a flat are, effectively, puzzlepieces of the big image, and embodiments of the full-resolutionlight-field rendering method reproduce the big image by bringing themicroimages sufficiently close together. The big image may also bereproduced by enlarging the pieces so that features from any given piecematch those of adjacent pieces. Assembling the resized pieces reproducesexactly the high-resolution image.

In either of these approaches, the individual pieces may overlap. FIG.15 illustrates a lens circle (or microimage) of diameter D and a patchof size m₁×m₂, where at least one of m₁ and m₂ is an integer greaterthan or equal to 2. Some embodiments of the full-resolution light-fieldrendering method avoid this overlapping by dropping all pixels outside asquare of size m₁×m₂, effectively cropping the microimage to an m₁×m₂square. Note that other embodiments may crop to other geometric shapes,such as a rectangle.

Conventional rendering methods do not reassemble pixels as describedabove; the conventional plenoptic camera algorithm produces one pixelper microlens for the output image. Embodiments of the full-resolutionlight-field rendering method, using the algorithm described above,produce a gain in resolution that is approximately equal to the numberof pixels m₁×m₂ in the original patches. That is, embodiments producem₁×m₂ pixels, instead of one pixel, per microimage

It has been shown above that the magnification M=n−1. It is also thecase that M=D/m. It therefore follows that:

$n = {1 + \frac{D}{m}}$

From the above, the distance (measured in number of focal lengths) tothe image plane in front of the microlens is related to D and m.

It is important to note that lenses produce acceptable images even whenthey are not exactly in focus. Additionally, out of focus images can bedeconvolved, or simply sharpened. For those reasons, the above analysisis actually applicable for a wide range of locations of the image plane.Even if not optimal, such a result is often a useful tradeoff.

The optics of the microlens as a camera is the main factor indetermining the quality of each microimage. Blurry images from opticaldevices may be deconvolved and the sharp image recovered to some extent.In order to do this, the effective kernel of the optical system shouldbe known. While there are limitations in this related to bit depth andnoise, embodiments may increase resolution up to m₁×m₂ times theresolution of a conventional plenoptic camera and conventional renderingmethod. Example embodiments have demonstrated a 27× increase ofresolution in one plane, and a 10× increase of resolution in anotherplane, when compared to conventional methods and apparatus, and withoutany deconvolution. Other embodiments may yield other increases inresolution when compared to conventional methods and apparatus.

Example Results

Some embodiments of a full-resolution radiance camera as describedherein may be implemented in film cameras. Embodiments may, for example,be implemented in large-format film cameras. An example large-formatfilm camera embodiment is illustrated in FIG. 8. One example embodimentmay, for example, be implemented in large-format film camera using a 135mm objective lens and 4×5 format film. A full-resolution radiance camerabased on a large-format film camera rather than on a digital camera maybe used for experimental purposes in order to avoid resolutionconstraint of digital sensors. However, film camera embodiments of thefull-resolution radiance camera design are practical and may havepractical applications. In conjunction with current high-resolutionscanners used to digitize captured images from negatives or prints,large-format film camera embodiments are capable of 1 gigapixel, or evenhigher, resolution for the flat (2D) representation of the 4D radiance(the raw flat).

A component of the full-resolution radiance camera is a microlens array.FIG. 16 shows a zoom into an example microlens array, and showsindividual microlenses and (black) chromium mask between themicrolenses. In one embodiment of a microlens array that may be used inthe example embodiment based on a large-format film camera, themicrolens array includes 146,000 microlenses of diameter 0.25 mm andfocal length 0.7 mm. A mechanism inside a 4×5 inch film holder of thelarge-format film camera holds the microlens array so that the flat sideof the glass base is pressed against the film. In one embodiment, thethickness of the microlens array is such that, when placed against thefilm, the microlenses are distance f from the film. Other configurationsof microlens arrays are possible, and the configuration of thelarge-format film camera makes it possible to easily changeconfigurations of microlenses by simply using a different microlensarray. Microsheets of glass may be used in the assembly as spacers orshims between the microlens array and the film to increase the distancefrom the microlenses and the film to be greater than f (e.g., 4/3 f). Anexample thickness of a microsheet that may be used is 0.23 mm. Insertingmicrosheet glass provides spacing in a rigorously controlled manner. Inone embodiment, additional spacing may be created by adding a singlemicrosheet between the film and the microlenses in order to displace themicrolenses by an additional ⅓ f, approximately 0.2 mm from the sensor.Additional microsheets may be added to provide additional spacing.

Experiments may be conducted both with and without inserting microsheetsof glass as spacers or shims between the microlens array and the film inthe example film camera used for testing. In both cases, the focallength of the microlenses is f=0.700 mm. The spacing in two experimentalconditions differ as follows:

-   -   b=0.71 mm so that n=71 and M=70, which is made possible directly        by the thickness of glass of the microlens array assembly        itself; and    -   b=0.94 mm based on microsheet glass between microlens array and        film. As a result, n=3.9 (almost 4) and M=3, approximately.

High-Resolution Rendering Methods and Results

FIGS. 17 through 20 are used to illustrate experimental results fromapplying the full-resolution rendering method to flats captured with theexample full-resolution radiance camera based on a large-format filmcamera described above. In particular, the operation of rendering inboth the telescopic case and the binocular case is illustrated anddescribed.

The original, unrendered flat was generated by capturing the image onfilm using the example radiance camera based on a large-format filmcamera, and digitizing the image via a scanning process using ahigh-resolution scanner. A portion of the digitized flat is shown inFIG. 17. After digitization, the full original flat is 24,862×21,818pixels, of which 2,250×1,950 pixels are shown in FIG. 17. Theapproximate region of the original flat extracted to produce FIG. 17 isshown by small solid white rectangle in FIG. 18C.

Output images rendered from the flat using conventional renderingmethods are shown in FIGS. 18A through 18C. The entire flat was renderedwith the conventional method, resulting in a 408×357 pixel image. FIG.18A is rendered at 300 ppi, while FIG. 18C is rendered at 72 ppi. At 300ppi, the image is only about 1 inch by 1 inch. FIG. 18B shows a 27×magnification of a crop of the curb area from the 300 ppi image in FIG.18A. The solid white rectangle in FIG. 18C shows the region from thelight-field shown in FIG. 17. The dashed white rectangle in FIG. 18Cshows a region that is rendered according to an embodiment of thefull-resolution light-field method as shown in FIGS. 19 and 20.

FIG. 19 show a full-resolution rendering of the experimentallight-field, rendered assuming the telescopic case according to oneembodiment of the full-resolution light-field rendering method describedherein. This region of the image is shown by the dashed white rectanglein FIG. 18C. For this rendering, the scaling-down factor was taken to beapproximately 2.4, so that the full-resolution rendered image measured11016×9666, i.e., over 100 megapixels. Even though the image is at 300dpi, only a 2,250×1,950 region is shown in FIG. 19. The image iswell-focused at full-resolution in the region of the house, but notwell-focused on the tree branches.

FIG. 20 shows a full-resolution rendering of the experimentallight-field, rendered assuming the binocular case according to oneembodiment of the full-resolution light-field rendering method describedherein. This region of the image is shown by the dashed white rectanglein FIG. 18C. Note that, in contrast to the image in FIG. 20, this imageis well-focused at full-resolution in the region of the tree branchesbut not well-focused on the house.

FIG. 21 is a flow chart illustrating how light is directed within aradiance camera according to one embodiment. As indicated at 700, lightfrom a scene is received at the main lens of a radiance camera. FIGS. 6,7 and 8 illustrate example radiance cameras. As indicated at 702, thereceived light is refracted by the main lens to an image plane. Asindicated at 704, an array of microlenses, the microlenses of which arefocused on the image plane, refracts light from the image plane onto aphotosensor located, relative to the microlenses, at a distance that isa multiple of the focal length f of the microlenses. For example, thedistance between the microlenses and the photosensor may be ¾ f, 4/3 f,5/3 f, 1.5 f, and so on. As indicated at 706, different views of theimage plane, refracted by the microlenses onto the photosensor, arecaptured at different regions of the photosensor to generate a flat,which is a 2D representation of the 4D light-field. In some embodiments,the photosensor may be a device configured to digitally capture lightsuch as a CCD, while in other embodiments the photosensor may beconventional film. As indicated at 708, the captured flat may berendered to produce a final high-resolution image, or images, of thescene, for example using a full-resolution light-field rendering methodas described in FIG. 22. For flats captured on conventional film, theflat may be digitized to generate a digitized flat before rendering.

FIG. 22 is a flowchart of a full-resolution light-field rendering methodaccording to one embodiment. As indicated at 800, a flat captured by aradiance camera may be obtained (see, e.g., FIG. 9 for an example ofwhat such a flat may look like to a human observer). As indicated at802, microimages in areas of the flat may be examined (manually orautomatically, as described below) to determine the direction ofmovement of edges in the microimages relative to a direction of movementof the algorithm. At 804, if it is determined that edges in microimagesof an area are moving relative to the microimage centers in the samedirection as the direction of movement, the microimages in that area maybe inverted relative to their individual centers. If the edges are notmoving relative to the microimage centers in the same direction as thedirection of movement (i.e., if the edges are moving in the oppositedirection as the direction of movement), then the microimages in thearea are not inverted.

In some embodiments, examination of the microimages to determine thedirection of movement of edges may be performed manually by a user via auser interface. The user may mark or otherwise indicate areas that theuser determines need be inverted via the user interface. In someembodiments, examination of the microimages to determine the directionof movement of edges may be performed automatically in software. In someembodiments, an automated software method may examine the microimages todetermine noise in the microimages, for example using a Fouriertransform to detect peaks at certain frequencies. An excessive amount ofnoise in an area of the final rendered image may indicate thatmicroimages in that area are flipped, and thus need to be inverted.Microimages that include noise over a specified threshold may be markedto be inverted.

As indicated at 806, the microimages may each be cropped to produce anm₁×m₂ subregion or crop of each microimage, where at least one of m₁ andm₂ is an integer greater than two. As indicated at 808, the subregionsor crops from the microimages may be appropriately assembled to producea final high-resolution image of the scene.

In some embodiments, instead of cropping the microimages and assemblingthe subregions generated by the cropping, the microimages themselves maybe appropriately assembled to produce a final high-resolution image ofthe scene. Thus, in these embodiments, element 806 is not performed; at808, the microimages are assembled to produce an output image. Inassembling the microimages, overlapping portions of adjacent microimagesmay be merged, blended, or otherwise handled.

In some embodiments, two or more images rendered from a flat accordingto rendering methods described herein may be combined to produce ahigher-quality output image. For example, in some embodiments, themicroimages in a flat may all be inverted, and the inverted microimagesappropriately assembled to produce a first intermediate image. A secondintermediate image may be generated without inverting the microimagesprior to assembling. The two intermediate images may then be combined toproduce a higher-quality output image. The combination of the two imagesmay be performed manually by a user via a user interface, for exampleusing a selection tool to select portions of an image to be combinedwith the other image, or alternatively may be performed automatically insoftware, for example using a noise detection technique as describedabove to find excessively noisy regions of one or both intermediateimages. As an example, when combining the images, the user may manually(or software may automatically) select areas in one intermediate imagethat are of higher quality than the same areas in the other image, andthen combine the selected areas with the other image to produce anoutput image that includes the highest quality portions of the twointermediate images. In some embodiments, a map (e.g., a bitmap) may begenerated that indicates areas of each image that are to be included inthe output image, and then the output image may be generated from thetwo intermediate images according to the map. In some embodiments, morethan two intermediate images may be generated, and a similar method maybe used to generate a higher-quality output image from the intermediateimages.

FIG. 25 is a flowchart of a full-resolution light-field rendering methodin which multiple images are rendered from a flat and combined toproduce a final high-resolution output image, according to someembodiments. As indicated at 1100, a flat captured by a radiance cameramay be obtained (see, e.g., FIG. 9 for an example of what such a flatmay look like to a human observer). As indicated at 1102, the pluralityof microimages may be assembled to produce a first high-resolutionimage. As indicated at 1104, each of the microimages may be invertedrelative to their respective centers to produce a plurality of invertedmicroimages. As indicated at 1106, the plurality of inverted microimagesmay be assembled to produce a second high-resolution image. As indicatedat 1108, the first high-resolution image may be combined with the secondhigh-resolution image to produce a final high-resolution image. Thecombination of the two images may be performed manually by a user via auser interface, for example using a selection tool to select portions ofan image to be combined with the other image, or alternatively may beperformed automatically in software, for example using a noise detectiontechnique as described above to find excessively noisy regions of one orboth intermediate images.

FIG. 26 shows an example full-resolution rendering of a light-field inwhich foreground and background portions of the example images shown inFIGS. 19 and 20 have been combined to produce a higher-quality outputimage. In FIG. 26, the foreground portion (the tree) of FIG. 19 has beenreplaced with the corresponding foreground portion of FIG. 19.

In some embodiments, multiple images may be rendered from a flataccording to rendering methods described herein, using different valuesfor m₁ and/or m₂ to crop the microimages before assembling the crops.This may produce multiple images with different visual quality. Forexample, assuming a square crop is to be made (i.e., m₁=m₂), someembodiments may be configured to perform the rendering using values form₁ and m₂ in a specified range, for example from 5 to 10 inclusive toproduce 6 output images, from 5 to 20 to produce 16 output images, andso on. One or more images may then be selected from among the multiplerendered images according to the quality of the images as outputimage(s). The selection may be performed manually, for example by a uservia a user interface, or alternatively may be performed automatically insoftware, for example using a noise detection technique as describedabove to select images with lower levels of noise in one or morefrequencies. Alternatively, two or more of the images may be selectedand combined to generate a higher-quality output image. The combinationof the images may be performed manually or automatically.

In some embodiments, inversion and cropping of microimages may becombined in a single automatic operation. For example, in someembodiments, a software module or modules configured to perform bothinversion and cropping of microimages in a flat or in a specified areaof a flat may have (m₁, m₂) as input parameters (or, alternatively, aninput parameter m if the crop is to be a square and thus m₁=m₂). Anegative value for (m₁, m₂) may be used to indicate that the microimagesin the input flat or area are to be inverted, with a positive value for(m₁, m₂) indicating that the microimages are not to be inverted. Othermethods to indicate whether microimages are to be inverted may be used.

In some embodiments, inversion and cropping of microimages may beperformed in various combinations on an input flat to render multiplerendered images according to the combinations. For example, in oneembodiment, some images may be rendered using a range of values for m₁and m₂ as described above while also inverting the microimages, whileother images may be rendered using a range of values for m₁ and m₂ asdescribed above in which the microimages are not inverted. One or moreof the rendered images may then be manually or automatically selected asoutput image(s). Alternatively, two or more of the rendered images maybe combined as previously described (see, e.g., FIG. 26) to produce anoutput image.

FIG. 23 illustrates a full-resolution light-field rendering modulerendering a high-resolution image from a flat captured, for example, bya radiance camera, according to one embodiment. Light-field renderingmodule 920 may implement a full-resolution light-field rendering methodas described in FIG. 22. FIG. 24 illustrates an example computer systemon which embodiments of light-field rendering module 920 may beimplemented. In some embodiments of a radiance camera, light-fieldrendering module 920 may be implemented in the camera, e.g. in captureddata processing module 260 of radiance camera 200 illustrated in FIG. 7.Referring to FIG. 23, light-field rendering module 920 receives an inputflat 910 captured by a radiance camera, such as one of the embodimentsof full-resolution radiance cameras described herein. Example portionsof a flat as may be captured by an example embodiment of a radiancecamera are illustrated in FIGS. 9 and 17. Light-field rendering module920 then processes the input image 910 according to the full-resolutionlight-field rendering method described herein. Light-field renderingmodule 920 generates as output a high-resolution image 930. FIGS. 19 and20 illustrate example high-resolution images that may be rendered andoutput by light-field rendering module 920. Output image 930 may, forexample, be stored to a storage medium 940, such as system memory, adisk drive, DVD, CD, etc. The dashed line from input image 910 tostorage medium 940 indicates that the original (input) flat 910 may alsobe stored. The dashed line from storage medium 940 to light-fieldrendering module 920 indicates that stored images may be obtained andprocessed by module 920.

In some embodiments, light-field rendering module 920 may provide a userinterface 922 via which a user may interact with the module 920, forexample to specify or otherwise manage input flats 910 and output images930 as described herein. In some embodiments, examination of microimagesto determine the direction of movement of edges may be performed by auser via the user interface 922. The user may mark or otherwise indicateareas that the user determines need be inverted via the user interface922. In some embodiments, examination of the microimages to determinethe direction of movement of edges may be performed automatically bylight-field rendering module 920. Some embodiments may allow eithermanual or automatic examination and detection, or a combination thereof,to be used. The user interface 922 may also provide tools whereby a usermay specify areas of two or more rendered images that are to be combinedto produce a higher-quality output image.

In one embodiment of a full-resolution light-field rendering methodimplemented in a full-resolution light-field rendering module 920, thetime required to render an image is proportional to the number ofmicrolenses times the number of pixels sampled under each microlens. Inother words, the time required to render an image is directlyproportional to the size of the output image 930.

Example System

Embodiments of a full-resolution light-field rendering module asdescribed herein may be executed on one or more computer systems, whichmay interact with various other devices. One such computer system isillustrated by FIG. 24. In the illustrated embodiment, computer system1000 includes one or more processors 1010 coupled to a system memory1020 via an input/output (I/O) interface 1030. Computer system 1000further includes a network interface 1040 coupled to I/O interface 1030,and one or more input/output devices 1050, such as cursor control device1060, keyboard 1070, audio device 1090, and display(s) 1080. In someembodiments, it is contemplated that embodiments may be implementedusing a single instance of computer system 1000, while in otherembodiments multiple such systems, or multiple nodes making up computersystem 1000, may be configured to host different portions or instancesof embodiments. For example, in one embodiment some elements may beimplemented via one or more nodes of computer system 1000 that aredistinct from those nodes implementing other elements.

In various embodiments, computer system 1000 may be a uniprocessorsystem including one processor 1010, or a multiprocessor systemincluding several processors 1010 (e.g., two, four, eight, or anothersuitable number). Processors 1010 may be any suitable processor capableof executing instructions. For example, in various embodiments,processors 1010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 1010 may commonly,but not necessarily, implement the same ISA.

System memory 1020 may be configured to store program instructionsand/or data accessible by processor 1010. In various embodiments, systemmemory 1020 may be implemented using any suitable memory technology,such as static random access memory (SRAM), synchronous dynamic RAM(SDRAM), nonvolatile/Flash-type memory, or any other type of memory. Inthe illustrated embodiment, program instructions and data implementingdesired functions, such as those described above for embodiments of afull-resolution light-field rendering module are shown stored withinsystem memory 1020 as program instructions 1025 and data storage 1035,respectively. In other embodiments, program instructions and/or data maybe received, sent or stored upon different types of computer-accessiblemedia or on similar media separate from system memory 1020 or computersystem 1000. Generally speaking, a computer-accessible medium mayinclude storage media or memory media such as magnetic or optical media,e.g., disk or CD/DVD-ROM coupled to computer system 1000 via I/Ointerface 1030. Program instructions and data stored via acomputer-accessible medium may be transmitted by transmission media orsignals such as electrical, electromagnetic, or digital signals, whichmay be conveyed via a communication medium such as a network and/or awireless link, such as may be implemented via network interface 1040.

In one embodiment, I/O interface 1030 may be configured to coordinateI/O traffic between processor 1010, system memory 1020, and anyperipheral devices in the device, including network interface 1040 orother peripheral interfaces, such as input/output devices 1050. In someembodiments, I/O interface 1030 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 1020) into a format suitable for use byanother component (e.g., processor 1010). In some embodiments, I/Ointerface 1030 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 1030 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. In addition, in someembodiments some or all of the functionality of I/O interface 1030, suchas an interface to system memory 1020, may be incorporated directly intoprocessor 1010.

Network interface 1040 may be configured to allow data to be exchangedbetween computer system 1000 and other devices attached to a network,such as other computer systems, or between nodes of computer system1000. In various embodiments, network interface 1040 may supportcommunication via wired or wireless general data networks, such as anysuitable type of Ethernet network, for example; viatelecommunications/telephony networks such as analog voice networks ordigital fiber communications networks; via storage area networks such asFibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 1050 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or retrieving data by one or more computer system 1000.Multiple input/output devices 1050 may be present in computer system1000 or may be distributed on various nodes of computer system 1000. Insome embodiments, similar input/output devices may be separate fromcomputer system 1000 and may interact with one or more nodes of computersystem 1000 through a wired or wireless connection, such as over networkinterface 1040.

As shown in FIG. 24, memory 1020 may include program instructions 1025,configured to implement embodiments of a full-resolution light-fieldrendering module as described herein, e.g. a direct lighting module 300,an indirect lighting module 800, or a global illumination renderingmodule 900, and data storage 1035, comprising various data accessible byprogram instructions 1025. In one embodiment, program instructions 1025may include software elements of embodiments of a full-resolutionlight-field rendering module as illustrated in the above Figures. Datastorage 1035 may include data that may be used in embodiments. In otherembodiments, other or different software elements and data may beincluded.

Those skilled in the art will appreciate that computer system 1000 ismerely illustrative and is not intended to limit the scope of afull-resolution light-field rendering module as described herein. Inparticular, the computer system and devices may include any combinationof hardware or software that can perform the indicated functions,including computers, network devices, internet appliances, PDAs,wireless phones, pagers, etc. Computer system 1000 may also be connectedto other devices that are not illustrated, or instead may operate as astand-alone system. In addition, the functionality provided by theillustrated components may in some embodiments be combined in fewercomponents or distributed in additional components. Similarly, in someembodiments, the functionality of some of the illustrated components maynot be provided and/or other additional functionality may be available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 1000 may be transmitted to computer system1000 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Accordingly, the present invention may bepracticed with other computer system configurations.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc. As well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent example embodiments of methods. The methods may be implementedin software, hardware, or a combination thereof. The order of method maybe changed, and various elements may be added, reordered, combined,omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended that the invention embrace all such modifications and changesand, accordingly, the above description to be regarded in anillustrative rather than a restrictive sense.

1.-24. (canceled)
 25. A computer-readable storage medium storing programinstructions, wherein the program instructions are computer-executableto implement: obtaining a flat comprising a plurality of separateportions of an image of a scene, wherein each of the plurality ofseparate portions is in a separate region of the flat, wherein each ofthe plurality of separate portions comprises a plurality of pixels, andwherein the flat is a 2D representation of a 4D light-field thatcaptures both spatial and angular information of the scene; assemblingthe plurality of separate portions to produce a first high-resolutionimage of the scene, wherein at least two pixels from each portion isincluded in the first high-resolution image; inverting each of theplurality of separate portions relative to their respective centers toproduce a plurality of inverted portions; assembling the plurality ofinverted portions to produce a second high-resolution image of thescene, wherein at least two pixels from each inverted portion isincluded in the second high-resolution image; and combining the firsthigh-resolution image with the second high-resolution image to produce afinal high-resolution image of the scene.